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This grant funded research on human-computer interaction design and 
analysis techniques, using future ATC environments as a testbed. The basic 
approach was to model the nominal behavior of both the automated and hu- 
man procedures and then to apply safety analysis techniques to these models. 

Our previous modeling language, RSML, had been used to specify the sys- 
tem requirements for TCAS II for the FAA. Using the lessons learned from 
this experience, we designed a new modeling language that (among other 
things) incorporates features to assist in designing less error-prone human 
computer interactions and interfaces and in detecting potential HCI prob- 
lems, such as mode confusion. The new language, SpecTRM-RL, uses “in- 
tent” abstractions, based on Rasmussen’s abstraction hierarchy, and includes 
both informal (English and graphical) specifications and formal, executable 
models for specifying various aspects of the system. One of the goals for our 
language was to highlight the system modes and mode changes to assist in 
identifying the potential for mode confusion. 

Three published papers resulted from this research (see attached). The 
first builds on the work of Degani on mode confusion to identify aspects 
of the system design that could lead to potential hazards. We defined and 
modeled modes differently than Degani and also defined design criteria for 
SpecTRM-RL models. Our design criteria include the Degani criteria but 
extend them to include more potential problems. In a second paper, Leveson 
and Palmer showed how the criteria for indirect mode transitions could be 
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applied to a mode confusion problem found in several ASRS reports for the 
MD-88. 

In addition, we defined a visual task modeling language that can be used 
by system designers to model human-computer interaction. The visual mod- 
els can be translated into SpecTRM-RL models, and then the SpecTRM-RL 
suite of analysis tools can be used to perform formal and informal safety 
analyses on the task model in isolation or integrated with the rest of the 
modeled system. 

We had hoped to be able to apply these modeling languages and analysis 
tools to a TAP air/ground trajectory negotiation scenario, but the develop- 
ment of the tools took more time than we anticipated. 
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Introduction 

Advanced automation has been accompanied, particu- 
larly in aircraft, with a proliferation of modes, where 
modes define mutually exclusive sets of system behavior. 
The new mode-rich systems provide flexibility and en- 
hanced capabilities, but they also increase the need for 
and difficulty of maintaining mode awareness. While au- 
tomation has eliminated some types of operator mode- 
awareness errors, it has also created the potential for 
new types of mode- related problems [SW95]. 

After studying accidents and incidents in the new, 
highly automated aircraft, Sarter and Woods have con- 
cluded that certain errors are non- random and pre- 
dictable [SW95]: They are the regular and predictable 
consequences of a variety of identifiable factors. Al- 
though these errors are accentuated by poor interface 
design and gaps or misconceptions in the user’s mental 
model of the system, an important factor is inconsistent 
automation behavior. 

Sarter and Woods have identified some of these pre- 
dictable error forms. Leveson et. al. [LPS97] and De- 
gani [Deg96] have defined taxonomies of automation fea- 
tures that lead to mode confusion. This paper describes 
an approach to dealing with mode confusion errors by 
first modeling blackbox software behavior and then us- 
ing analysis methods and tools to assist in searching the 
models for predictable error forms, i.e., for features that 
contribute to operator mistakes. The analysis results 
can be used to redesign the automation, to change op- 
erator training and procedures, or to design appropriate 
human-computer interfaces to help avoid mistakes. 

The approach requires a model of the blackbox behavior 
that is both formal and easily readable and reviewable 
by humans. The models we use are part of the software 
specifications in a methodology called SpecTRM (Speci- 
fication Tools and Requirements Methodology) and thus 
the analysis is done directly on the system requirements 
specification and does not require extra modeling ef- 
fort. SpecTRM includes a suite of analysis tools to 
detect errors and potentially hazardous behavior early 
in system development when tradeoffs and changes can 
more easily be made. In addition to providing design 


guidance, this approach might provide a way of “mea- 
suring” or evaluating the cognitive demands involved 
in working with specific automated devices. Hansman 
has suggested that automation complexity be defined in 
terms of the predictability of the automation behavior 
[Hans97]. This predictability can potentially be evalu- 
ated on the formal SpecTRM-RL (SpecTRM Require- 
ments Language) models. 

The rest of the paper provides more information about 
the approach and illustrates its use on a commonly re- 
ported mode confusion error called a “kill-the-capture” 
bust. 

Mode Confusion Analysis 

Most accidents related to software behavior can be 
traced back to errors or omissions in the software re- 
quirements, not to implementation or coding errors 
[Lev95, Lut93]. Although a great deal of effort has been 
expended in software engineering on finding software 
design and implementation errors, much less has been 
accomplished in terms of validating requirements spec- 
ifications beyond executing them for a few test cases 
or showing the consistency of a formal specification 
with various properties of the underlying mathemati- 
cal model [HL96, HLK95]. Most of the specfication er- 
rors and omissions that lead to accidents are unlikely to 
be found using these techniques. The testing of any 
complex software is necessarily very incomplete, and 
consistency with a mathematical model does not im- 
ply consistency with required properties of a real world 
application. 

To deal with this problem, we have specified a set of 
criteria for completeness and correctness of blackbox 
process-control requirements specifications that are re- 
lated to safety [JLHM91, Lev95]. These criteria were 
derived using real accidents and industrial experience 
with process-control software, and they have been vali- 
dated by experimental application to the NASA Gallileo 
and Voyager software [Lut92, Lut93] and through indus- 
trial use. We are contining to extend the criteria, most 
recently with the goal of reducing mode confusion er- 
rors, and to validate them on real software [MLR97]. 
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Figure 1: Simple Control Loop Model 
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To apply the criteria, a blackbox state- machine model of 
the automation behavior is required. Blackbox require- 
ments specifications do not contain information about 
internal design (the software design if the automated 
controller is a computer) but are written strictly in 
terms of externally visible inputs and outputs and the 
effects of these on a model of the process being con- 
trolled (see Figure 1). The process model is based on: 

1. Current process state inferred from measured vari- 
ables, 

2. Past measured and inferred process states and vari- 
ables, 

3. Past outputs to actuators, and 

4. Prediction of future states of the controlled process. 

Accidents related to requirements (behavioral) specifi- 
cation occur when the internal model of the process be- 
comes inconsistent with the state of the controlled pro- 
cess. This inconsistency may result from an incorrect 
model being specified originally (e.g., the model does 
not include basic required behavior for unusual or in- 
frequently occurring cases) or from the modeled system 
state being updated incorrectly during execution, per- 
haps as a result of incorrect input from the sensors. 

To define criteria related to mode confusion, we need to 
add a model of the controller-software interface to the 
automated controller. We also need to consider the su- 
pervisors 3 internal models of the expected behavior and 
state of the process and of the automated controller (see 
Figure 2). Accidents in this extended model may result 
from any of these models being incorrect or becoming 
inconsistent with the true state of the controlled pro- 
cess, the automated controller, or the supervisory inter- 
face (the human-computer interface). That is, accidents 
may result if any of the models are or become inconsis- 
tent with the state of the thing they are modeling and 
decisions or actions are made on the basis of the incor- 
rect model. Criteria for correctness and safety can be 
specified in terms of these formal models and checked 
for particular system specifications. 
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Figure 2: Modified Model to Account for Operator Er- 
ror and Mode Confusion. 


Of course, we are not suggesting that it is possible to 
specify human mental models. Each person may have a 
different mental model of the system and the automa- 
tion, and these may change over time within the same 
person. In fact, operators have been found to be able to 
function with multiple and inconsistent models [Luc87]. 
However, it is possible to state some high-level abstrac- 
tions about required features of correct operator men- 
tal models — for example, that particular actions on the 
part of the operator will result eventually in particular 
changes in the automation and/or the system. 

Note that we assume here that the operator’s models are 
correct. This assumption will obviously not always be 











true. However, our approach involves first eliminating 
hazards for the ideal case. Then various types of hazard 
analysis can be used to determine which types of erro- 
neous models will have the most serious consequences. 
The resulting information can be used for automation 
design, interface design, and operator training. 

A previous paper described six categories of potential 
design flaws that can lead to mode confusion errors: 
interface interpretation flaws, inconsistent behavior, in- 
direct mode changes, operator authority limits, unin- 
tended side effects, and lack of appropriate feedback 
[LPS97]. The rest of this paper shows an example of 
this approach for one particular common cause of mode 
confusion error, i.e., indirect mode changes. The ba- 
sic criteria and analysis technique is being specified for- 
mally [Lev97], but we include only an informal descrip- 
tion here. 

Indirect Mode Change Example 

Indirect mode changes occur when the automation 
changes mode without an explicit instruction by the op- 
erator. Such transitions may be triggered on conditions 
in the controller (such as preprogrammed envelope pro- 
tection) or sensor input about the state of the controlled 
system (such as achievement of a target value). Indirect 
mode transitions create the potential for mode confu- 
sion and inadvertent activation of modes by the human 
controller. For example, the human controller may not 
update his or her models of the state of the process and 
the state of the automation and, based on these now in- 
correct models, issue an incorrect control command or 
fail to issue a required command. 

An example of an accident that has been attributed to 
an indirect mode change occurred while an A-320 was 
landing in Bangalore. In this case, the pilot selection 
of a lower altitude while the automation was in the al- 
titude ACQUISITION mode resulted in the activation 
of the OPEN DESCENT mode. It has been speculated 
that the pilots did not notice the mode annunciation 
because the indirect mode change occurred during ap- 
proach when the pilots were busy and they were not 
expecting the change [SW95]. Another example of such 
an indirect mode change in the A-320 automation in- 
volves an automatic mode transition triggered when the 
airspeed exceeds a predefined limit. For example, if the 
pilot selects a very high vertical speed that results in 
the airspeed decreasing below a particular limit, the au- 
tomation will change to the open climb mode, which 
allows the airplane to regain speed. 

Palmer has described another example of a common 
indirect mode transition problem called a “kill-the- 
capture bust” that has been noted in many ASRS re- 
ports [Pal96]. Here we show the relevant parts of a 
SpecTRM-RL specification of the MD-88 control logic 
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Figure 3: Flight Mode Annunciator (FMA) Displays for 
the Example Incident 


and describe how the problem can be detected and fixed. 

In the incident, the crew had just made a missed ap- 
proach and had climbed to and leveled at 2,100 feet. 
Figure 3 shows the sequence of Flight Mode Annun- 
ciator (FMA) values during the incident. The crew 
received the clearance to . . climb now and maintain 
5,000 feet The Captain set the desired altitude 

to 5,000 feet, set the autopilot pitch mode to verti- 
cal speed with a value of approximately 2,000 feet per 
minute and the autothrottle to SPD mode with a value 
of 256 knots (Figure 3(c) and (d)). Climbing through 
3,500 feet, the Captain called for flaps up, and at 4,000 
feet he called for slats retract and pushed the IAS but- 
ton (Figure 3(f)). The pitch mode became IAS, and 
the autothrottle went to CLAMP mode. At this point, 
altitude capture was still armed. Three seconds later, 
the autopilot automatically switched to altitude cap- 
ture mode. The arm window went blank, and the pitch 
window showed ALT CAP (Figure 3(g)). A tenth of a sec- 
ond later, the Captain adjusted the vertical speed wheel 
to a value of about 4,000 feet per minute. This speed 
adjustment caused the pitch autopilot mode to switch 
from altitude capture to vertical speed (Figure 3(h)). 
Climbing through 4,500 feet, the FMA was as shown 
in Figure 3(h), and the approaching altitude light was 
on. As the altitude passed through 5,000 feet at a verti- 
cal velocity of about 4,000 feet per minute, the Captain 
remarked, “Five thousand. Oops, it didn’t arm.” He 
pushed the alt HOLD button and switched off the au- 
tothrottle. The aircraft continued to climb to about 




5,500 feet and the ALTITUDE-ALTITUDE voice warning 
sounded repeatedly. 

To identify and fix the problem, we use a formal model. 
A SpecTRM-RL model has two parts: a graphical model 
of the state machine and a specification of the logic on 
the transitions. Figure 4 shows part of the graphical 
SpecTRM-RL state machine model of the MD-88 verti- 
cal control logic needed to understand the incident and 
how to fix the software to avoid it. In order to keep the 
model small enough to fit in the paper, only parts of 
it are shown but during system engineering a complete 
model would be constructed. The graphical model has 
three main parts: the input-output interface (where the 
supervisory interface is one part), the operating modes 
of the automation itself (in this case the autoflight sys- 
tem), and the process model which includes both the 
process (aircraft) operating modes and models of the 
aircraft components. 

In the supervisory interface, square boxes denote in- 
puts and outputs having finite state values. Circles rep- 
resent numbers. Note that this model represents the 
automated controller’s view of the state of the inter- 
face, not necessarily the real state of the controls and 
displays. A complete safety analysis would evaluate if 
and how discrepancies between the two could occur and 
also whether such discrepancies could lead to hazardous 
system states. 

The state transition logic is specified in SpecTRM-RL 
using a form of logic tables we call and/or tables. A 
transition can be taken if any of the columns of the table 
evaluates to true. A column evaluates to true if all the 
(non-blank) rows in a column are true. Figure 5 shows 
the relevant transition logic for the example. 

The problem occurs because the transition to ALT CAP 
mode results in a transition of the capture mode to UN- 
ARMED before the altitude has actually been acquired. 
Although this is annunciated to the pilot by the Arm an- 
nunciator changing to blank when pitch mode changes 
to ALT CAP, the absence of an indicator is well known 
to be an error-prone way to notify the pilot of a mode 
change. 

How could this be detected from examining the logic? 
In general, an indirect mode change is one that occurs 
without an explicit pilot action to change the mode. The 
vertical control logic for the example has three mode 
transitions that do not require direct pilot input: (1) 
the transition from ANY to ALT CAP, (2) the second col- 
umn of the transition from ANY to ALT HOLD, i.e, when 
the altitude is acquired and the pitch is in mode ALT 
CAP, and (3) the second column of the transition from 
ARMED to NOT ARMED. Each of these mode transitions 
is triggered by a change in a controlled system variable 
or by internal mode change within the automation. 


We will assume that the pilot’s mental model includes 
a cause and effect relationship between arming the al- 
titude capture and eventually (although it may not be 
immediately) acquiring that altitude and holding it: 

set altitude and pull alt alt hold. 

Formal analysis will show that there is a path through 
the logic starting with the pilot pulling the altitude knob 
that does not result in the ALT HOLD state (specifically, 
this occurs when the automation is in the modes NOT 
armed and alt CAP and the pilot does something that 
changes the pitch mode, in this case adjusting the ver- 
tical speed wheel). 

One way to fix the problem is to change the transition 
logic to that shown in Figure 6. Note that although the 
second column of the transition table from ANY to ALT 
HOLD still does not require direct pilot input, the transi- 
tion is not indirect by our definition because it satisfies 
the pilot model of the transition logic above. The tran- 
sition from any to alt CAP is still indirect, but there is 
no longer a path through the vertical control logic that 
violates the expected cause and effect relation between 
arming the capture and capturing the altitude when it is 
acquired. We note that this solution may violate other 
goals or desired behaviors of the autoflight system — the 
designers would have to determine this when deciding 
what solution to use. In addition, a more sophisticated 
solution may be required, e.g., a hysteresis factor may 
need to be added to the mode transition logic to avoid 
too rapid or a ping-ponging” transitions between pitch 
modes. 

Finding indirect mode transitions does not mean the 
software must be changed. The identified criteria are 
simply clues for determining where to look for potential 
problems. The designers may decide that no real prob- 
lem exists and make no changes or they may decide not 
to change the automation but instead to make changes 
in the interface design or in pilot training. 

In general, it is not feasible to make all mode transitions 
direct in any sophisticated automated controller. The 
goal instead is to simplify the required pilot model of 
the automation behavior as much as possible. In this 
case, the pilot expects a direct mode transition from set- 
ting a target altitude and arming the altitude capture to 
eventually attaining capturing altitude, changing to ALT 
HOLD mode, canceling the ARM command. Any paths in 
the automation logic that will violate this assumption 
will be a source of potential mode confusion, even if the 
mode change is annunciated. Of course, the pilot may 
have more sophisticated knowledge of the automation 
logic and know that adjusting the vertical speed wheel 
will cancel the previously given altitude capture com- 
mand. However, ,this knowledge assumes a much more 
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Figure 4: Part of the Graphical State Machine Model of the MD-88 Vertical Control Logic 
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complex model of automation behavior on the part of 
the pilot and makes the automation behavior more dif- 
ficult to predict. The number of ASRS reports on this 
error leads us to believe that this assumption is not re- 
alistic. 

Several caveats are important here. First, we are only 
guessing at the MC-88 software logic on the basis of the 
observed behavior. The real software specifications or 
code would have to be examined to determine what logic 
is actually implemented. Second, making the change we 
have recommended may not be feasible (or correct) due 
to other unmodeled parts of the logic that depend on the 
armed mode: A real development project would have 
the entire logic modeled and would be able to make the 
appropriate tradeoffs and design decisions. 
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Abstract 

As control systems become more complex, the use of 
automated control has increased. At the same time, the 
role of the human operator has changed from primary 
system controller to supervisor or monitor. Safe de- 
sign of the human- computer interaction becomes more 
difficult. 

In this paper , we present a visual task modeling lan- 
guage that can be used by system designers to model 
human- computer interactions. The visual models can 
be translated into SpecTRM-RL , a blackbox specifica- 
tion language for modeling the automated portion of 
the control system. The SpecTRM-RL suite of analysis 
tools allow the designer to perform formal and infor- 
mal safety analyses on the task model in isolation or 
integrated with the rest of the modeled system. 

1 Introduction 

Increased complexity of control systems and advances 
in computer technology have combined to give automa- 
tion a more authoritative role in control systems. As a 
result, many of these control systems rely on both hu- 
man and automated controllers. For these controllers 
to interact effectively, the human-computer interaction 
must be carefully designed. 

We began looking at these issues while working 
with the Terminal Area Productivity (TAP) Project 
at NASA Ames [PPC97]. The TAP Project is design- 
ing terminal area procedures for air traffic using data 
links in addition to voice contact to communicate tra- 
jectories and routing information between the air traf- 
fic controller and the aircraft. 

Traditionally, HCI design has focused on the human 
user’s point of view: what functionality is needed to 
support the tasks the human must accomplish. There 
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have been numerous models for human task analysis 
developed to identify the knowledge and steps required 
to perform each human task. 

Unlike traditional task modeling methods that focus 
on analyzing specific aspects of the HCI design, such 
as user goals or knowledge representation, our model- 
ing technique has a more general focus. Our technique 
allows the designer to model the steps required to com- 
plete the task so she can study how these steps interact 
with the rest of the system. Specific environment cues 
required to complete a task are modeled as conditions 
on transitions between steps in a task. Depending on 
the visualization created to inspect the model, the an- 
alyst can focus on the knowledge needed throughout 
the task, on the steps required to complete the task or 
on some other model aspect of interest. 

Our modeling methodology takes a system-centric 
view compared to the human-centered view of other 
task analysis methods. With a tighter coupling be- 
tween the human and automated controllers in com- 
plex systems, the human controller must be viewed as 
a part of the entire system, therefore the interaction be- 
tween the human and the computer should be viewed 
in the context of the entire system. 

Our modeling technique focuses on formally analyz- 
ing a model of the controller’s tasks independently and 
in the context of the complete control system model. 
Throughout the evolution of our method, we had three 
goals: 

• To create a reasonable model of the actions of the 
human controller. 

• To develop a model that can be formally analyzed 
with respect to safety concerns. 

• To interface the model of the human controller tasks 
with formal models of the rest of the complex sys- 
tem. 



To realize these goals, we created a visual task 
modeling language that allows the analyst to easily 
represent the necessary information about the con- 
troller’s tasks. These task models can be translated 
into the blackbox requirements specification language 
SpecTRM-RL and analyzed using a suite of anal- 
ysis tools. SpecTRM-RL was developed to model 
all the components of control systems, therefore the 
SpecTRM-RL model of the controller tasks can be in- 
tegrated with the model of the other components in 
the control system. 

The rest of the paper is organized in the following 
way: Section 2 describes the SpecTRM tool suite and 
SpecTRM-RL modeling language; Section 3 describes 
our approach to achieving the goals we set for our- 
selves; Section 4 explains the example model that we 
use throughout the paper to illustrate our task model- 
ing methodology; Sections 5 and 6 describe the visual 
task modeling language and how these models are cre- 
ated; Section 7 demonstrates how the safety analysis 
tools can be applied to these models; Section 8 dis- 
cusses other work in the area of task modeling and 
analysis; and Section 9 discusses the contributions of 
this work and possible future questions to explore. 

2 SpecTRM 

The Software Safety Group at the University of Wash- 
ington has developed a methodology for software spec- 
ification called SpecTRM (Specification Tools and Re- 
quirements Methodology). Complex system develop- 
ment relies on multiple disciplines: system engineers, 
software engineers, human factors experts and ap- 
plication experts. SpecTRM takes a global system 
viewpoint to provide an environment to assist multi- 
disciplinary teams. With a single consistent model of 
the complex system, analysts from each discipline can 
focus on the aspect of the system that is of interest to 
them. 

The center of SpecTRM is SpecTRM-RL (SpecTRM 
Requirements Language), a formal requirements spec- 
ification language for modeling blackbox behavior of 
control systems [Lev98]. SpecTRM-RL supports a 
wide-range of problem-solving strategies and tasks dur- 
ing system development and evolution. 

Models written in SpecTRM-RL can be analyzed by 
the SpecTRM suite of tools. The SpecTRM tools were 
developed to detect errors and potentially hazardous 
behavior. The tool set is being expanded but currently 
includes: 

• Model execution 

• Automated formal analyses, such as consistency and 

completeness checks 


• Automated tools to help with model exploration 
during forward and backward analyses 

• Deviation analysis to test the robustness of system 
design to abnormal inputs 

• Visualization tools to allow a wide-range of views of 
the executing model 

These tools provide a flexible framework in which many 
complimentary analyses can be performed on a single 
model to help ensure total system safety. 

3 Approach 

We do not attempt to model erroneous human behav- 
ior, but limit our models to the expected controller 
behavior (both nominal and off-nominal) as defined in 
operational procedures. The models also include ex- 
ternal inputs representing qualities of the environment 
that give rise to the human controller’s decisions. 

In many cases, the exact triggering conditions of a 
task are not necessary to have a meaningful model of 
the system. Any external or internal conditions that 
are particularly salient to the executing task are ex- 
plicitly modeled while all other environmental effects 
are grouped into external conditions. 

We quickly found that SpecTRM-RL models were 
not the most effective way to specify human tasks for 
system designers and human factors experts. For one 
thing, it was difficult to separate nominal from off- 
nominal behavior using the SpecTRM-RL notation. It 
was also difficult to see communication flow between 
the different components in the model. These limita- 
tions in expressibility led us to the development of a 
visual modeling language that has the characteristics 
lacking in SpecTRM-RL but remains easily translat- 
able to SpecTRM-RL for analysis purposes. 

4 Handoff Procedure Example 

Throughout this paper, we will be using our model 
of a handoff procedure to illustrate our task modeling 
language. A “handoff”, or a change of aircraft con- 
troller, occurs whenever an aircraft is changing from 
one controlling sector to another. The handoff pro- 
cedure involves communication between the controller 
currently controlling the aircraft, the next controller 
to control the aircraft, and the pilot. Our model of 
this procedure includes the required tasks from each 
controller’s point of view and the pilot’s point of view. 
The handoff procedure model also includes a model of 
the radio used by the pilot to control the frequency to 
which she listens for controller communication. The 
radio component is included in the model because it 



is an integral part of the hardware/software portion of 
the system during the handoff procedure. 

We chose this model to demonstrate our model- 
ing and analysis technique because the model is com- 
plex enough to have interesting characteristics while 
remaining clear enough to be easily understood by 
readers who are not familiar with air traffic control. 
The handoff procedure consists of multiple components 
each with variety of tasks to complete and interesting 
interactions with other components. 

5 Description of Visual Modeling Lan- 
guage 

Figures 1, 2 and 3 show the visual model for current 
implementation of the handoff procedure in air traffic 
control systems. 

The key on Figure 1 displays the components of the 
visual modeling language. States are used to represent 
each step required to complete a higher-level task. To 
change from one state to the next in the model, a tran- 
sition must occur. Changing the current model state 
represents the completion of one subtask and the be- 
ginning of the next subtask. An event is the triggering 
condition for a transition and an action results from 
completing a transition. The default, or start, state 
is denoted by an arrow head on the left side of the 
state. For example, in the Current Controller model of 
Figure 1 the state Aircraft being controlled by current 
controller is the start state. The model can transition 
to the state Initiating handoff if the event Conditions 
for handoff occur fires. In this example, there is no 
resulting action from the transition. 

Color is used in the model to differentiate between 
events and actions. The events triggering a transition 
are shown in blue text above the transition, while any 
actions resulting from the transition are shown in red 
text beneath the transition. A green outline around 
event or action text denotes that this is a communi- 
cation point between two entities in the model. For 
example, in Figure 1 there is a green outline around 
the action Initiate Handoff in the Current Controller 
model and a green outline around the event Initiate 
Handoff in the Next Controller model. The green out- 
line denotes that the action of initiating a handoff in 
the Current Controller model causes a transition to 
occur in the Next Controller model. 

Task positioning is used to represent relationships 
among the steps in a task. The normative actions are 
seen on the main horizontal axis of each controller’s 
task model. Any non-normative behavior diverges 
from the main axis until the situation has been cor- 
rected and the normative procedure resumes. For ex- 


ample, in the Next Controller model of Figure 1, reject- 
ing the handoff is not the expected or nominal behavior 
of the Next Controller. The events, states and actions 
required to handle this sequence of subtasks are shown 
below the main axis of the Next Controller’s behaviors. 
When the off-nominal steps have been completed, the 
model can return to a state on the main horizontal axis 
to represent returning to the nominal behavior. 

The visual modeling language also allows the sys- 
tem designer to represent the relationship among the 
tasks that are being carried out by the model entities. 
To accurately represent the load on the human con- 
troller, the system designer must understand these re- 
lationships accurately. The system designer must un- 
derstand which tasks are sequential and which tasks 
can be performed in any order. The Pilot model, Fig- 
ure 2, shows how these task relationships are repre- 
sented. The action Issue frequency change must occur 
before the Pilot can perform the subtasks to change the 
radio frequency and read back the Next Controller’s 
frequency to the Current Controller. The branching in 
the transition arrows shows how changing the radio fre- 
quency and performing the read back are executed in 
some undetermined order. These tasks could be per- 
formed in parallel or in a sequential order chosen by 
the Pilot. The system designer must understand these 
possible interactions to ensure that the tasks do not 
overload the Pilot no matter what order she chooses 
to execute the tasks. 

6 Construction of Task Models 

The basis of the visual model is a task analysis. The 
task analysis identifies the major tasks of the con- 
troller components in the system then breaks these 
high level tasks into subtasks, down to the level of 
the key presses, voice communications, display cues, 
etc. involved in performing the task. From the task 
analysis, the visual model is created. The visual model 
easily represents the temporal relationships among the 
tasks carried out by a single component in the model 
and the relationships among multiple components in 
the model. 

The SpecTRM-RL model is created based on the 
visual model. The conversion is a straightforward pro- 
cess of converting the visual relationships to a text- 
based modeling language. Whereas the visual model- 
ing language uses entities, states, events and actions to 
represent the system, the SpecTRM-RL model decom- 
poses the system into components, operating modes, 
and input/output interfaces. A component is a por- 
tion of the system with a well-defined interface to the 
rest of the system. Each high-level system entity and 
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the external environment become a component in the 
SpecTRM-RL model. For the handoff procedure, the 
components in the SpecTRM-RL model are the Cur- 
rent Controller, the Next Controller, the Pilot, and 
the Radio. The states in the visual model translate 
to operating modes in the SpecTRM-RL model. For 
example, from the visual model shown in Figure 3 
the Radio component has two operating modes: Fre- 
quency 1 Active/ Frequency 2 Standby and Frequency 2 
Active/ Frequency 1 Standby. 

Translating the communication interfaces is slightly 
more difficult. The events in the visual model map to 
input interfaces for the SpecTRM-RL model compo- 
nents. If the event was marked as a communication 
point with another entity in the visual model (denoted 
by the green outline around the event label), the cor- 
responding input will come from another SpecTRM- 
RL component. Otherwise, the event will map to an 
input from the external environment. Actions in the 
visual model translate to the output interfaces for the 
SpecTRM-RL components. 

7 Analysis of Models 

The human procedure model, along with SpecTRM- 
RL models of the other parts of the system, can be used 
in the safety analysis of the human-computer interac- 
tion. The SpecTRM tool set currently allows model 
execution, various types of safety analysis, and visual- 
ization. 

7.1 Model Execution 

Because the task models are executable, the system 
designer can inspect the specified dynamic interaction 
between the system components, including both the 
operators and the automated components. In this way, 
procedural errors, possible inconsistencies in the pro- 
cedures, or incomplete procedural specifications can be 
detected. We found that model execution helped us to 
find several errors in the specification. 

7.2 Safety Analysis Tools 

The SpecTRM tool set has multiple fully automated or 
partially automated analyses that can be performed on 
the models to help identify possible unsafe aspects of 
the system requirements. Consistency and complete- 
ness analysis identifies inconsistencies in the specifica- 
tion and conditions not accounted for in the specifica- 
tion [HL96]. For example, the automated completeness 
check on this model found that the Pilot’s behavior is 
not completely specified. During the handoff proce- 
dure, the model does not show how the Pilot should act 
if she reads back an incorrect frequency. This incom- 
pleteness in the specification is likely to be the result 


of an oversight of the system designer as she builds the 
model, but the completeness check aids the analyst by 
highlighting these possible oversights. 

Deviation Analysis provides a way to evaluate 
the specification for robustness against incorrect in- 
puts [RL97a, RL97b]. The analyst denotes potential 
hazardous outputs that she wants to check for and hy- 
pothesizes deviations in the inputs, for example, that 
measured speed is lower than actual speed. The Devi- 
ation Analysis tool will determine whether the devia- 
tion can lead to a hazardous state and, if not, whether 
the hypothesized deviation plus other conditions could 
lead to a hazard. 

Our Backward Analysis tool allows the system de- 
signer to start from a hazardous state and work back- 
ward to determine if and how that state could be 
reached. Critical points in the design are identified 
that can be modified to avoid the hazardous state. For 
example, a backward analysis of the handoff procedure 
model found that the Pilot model can reach the haz- 
ardous state Pilot Tuned To Unknown Frequency. 

7.3 Visualization Tools 

Our IBToolKit (Interface Builder Tool Kit) allows the 
system designer to create visualizations that can be 
linked to a SpecTRM-RL model [Pin97]. In addition 
to simply showing the results of the model executing 
in a visual format, visualizations can be created that 
highlight specific system qualities. These types of visu- 
alizations can aid in the design of automated systems 
to maximize the strengths of both the human and au- 
tomated controllers. 

One example of a visualization that we created for 
the handoff procedure is shown in Figure 4. In this vi- 
sualization, the cognitive demands on the pilot during 
task execution are highlighted as the model executes. 
The states Pilot has not changed frequency and Fre- 
quency not read back are highlighted in red to denote 
the current state of the model. From this visualization, 
the system designer can see that the pilot had to detect 
a change in her environment in order to transition out 
of the Under firm control of current controller state. 

Another possible visualization would be a display of 
the cockpit. As the portions of the tasks are com- 
pleted by the pilot interacting with the cockpit con- 
trols, the corresponding areas of the cockpit could be 
highlighted. This visualization assists the system de- 
signer in determining whether the actions involved in 
the task support the cognitive processes required of the 
pilot. 



8 Related Work 

Task analysis models such as Hierarchical Task Anal- 
ysis (HTA) [AD67] focus on steps required by the hu- 
man to complete the given task. In HTA, tasks can 
be broken down into subtasks and there is no restric- 
tion on the level of decomposition. The view is very 
human-centered since this model was initially proposed 
to represent tasks for the purpose of training new users 
of applications. The focus is on the steps that must be 
completed by the human to accomplish a task with- 
out consideration for the operations of the computer. 
HTA is very flexible, but this flexibility often leads to 
ambiguity and inconsistency among models. 

Many task models have been based on the Goals, 
Operators, Methods and Selection Rules (GOMS) 
framework [CMN83]. In the GOMS methodology, the 
user’s goals are decomposed into subgoals that have op- 
erators (behaviors) and methods (sequences of behav- 
iors) associated with them. The goals are decomposed 
into four different levels of description. This model fo- 
cuses on error-free performance and models the tasks 
down to a very detailed description of the keystroke 
level of behavior. 

Task Analysis for Knowledge Descriptions (TAKD) 
takes a different approach to task modeling by focusing 
on the knowledge required by people to complete the 
tasks [DJ89]. The TAKD method allows the knowledge 
needed to complete a task to be represented by a multi- 
level abstraction that traces the high-level knowledge 
down to the low-level steps necessary to complete the 
task. Johnson et al. developed another task analysis 
approach, Task Knowledge Structures (TKS), based 
on TAKD [JJ91]. The TKS methodology represents 
tasks for the purpose of defining what tasks need to be 
supported by the system design. Unlike many other 
task models, TKS is a tool for designing systems, not 
just evaluating existing designs. 

Executable task models have been developed and 
used for modeling human-computer interaction. Yost 
used the Soar/TAQL to model the Sisyphus-93 
elevator-configuration task [Yos96]. To achieve an ex- 
ecutable model, Yost began with a knowledge-level 
model of the task, refined this model to a problem- 
space model, then created the symbol-level model that 
is executable. The symbol-level model of the task 
was executed to help understand what problem-solving 
steps are involved during the completion of the task. 
The Programmable User Model (PUM), another exe- 
cutable model, provides a constrained cognitive archi- 
tecture that can be programmed to simulate the user’s 
execution of some set of tasks using a proposed inter- 
face [YGS89]. PUMs are meant to aid the interface 


designer in creating more usable designs by highlight- 
ing psychological considerations. 

Joint human-computer controlled systems change 
the human controller’s function from an active con- 
troller to a supervisor and error handler. However, au- 
tomation design has not changed to support the human 
controller’s new role as a monitor of system behavior 
and backup controller in case of emergency [SW95]. 
The Operator Function Model (OFM) provides a tool 
to help improve automation design so that the human 
controller tasks are supported [Mit96]. The OFM de- 
scribes the required operator behavior by representing 
the operator functions, subfunctions and information 
needed by the operator for each activity. 

9 Contributions and Future Work 

We are in the process of testing this technique by mod- 
eling the controller tasks in the new air traffic control 
system being developed by the TAP Project at NASA 
Ames. We believe that the development of these tools 
will provide system designers and analysts with a more 
rigorous method of analyzing the assigned controller 
tasks for safety. The SpecTRM tool set provides au- 
tomated analysis capabilities to the systems designer 
while the visual modeling language and flexibility of 
the IBToolKit allows the system designer to make de- 
sign decisions with much better knowledge of the effect 
of these decisions on overall system safety. 

There is future work to be done in the areas of special 
analyses on these types of models. We would like to see 
if we can use the models to assist in task allocation. 
Another problem we would like to explore using the 
models is mode confusion. 

We would also like to continue exploring the flexi- 
bility in model analysis that is provided by the visual- 
ization tools. We are currently working on an interac- 
tive visualization of the ground procedures of the TAP 
FAST-system with datalink capabilities. This visual- 
ization to experiment with the use of an interactive 
model in system design. 
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Abstract 

Increased automation in complex systems has led 
to changes in the human controller’s role and to new 
types of technology-induced human error. Attempts 
to mitigate these errors have primarily involved giv- 
ing more authority to the automation, enhancing op- 
erator training , or changing the interface. While 
these responses may be reasonable under many circum- 
stances, an alternative is to redesign the automation 
in ways that do not reduce necessary or desirable func- 
tionality or to change functionality where the trade- 
offs are judged to be acceptable. This paper describes 
an approach to detecting error-prone automation fea- 
tures early in the development process while significant 
changes can still be made to the conceptual design of 
the system. The information about such error-prone 
features can also be useful in the design of the operator 
interface, operational procedures, or operator training. 


Introduction 

Today’s large, complex systems often incorporate 
both human and automated control and monitoring. 
These jointly controlled systems are starting to expe- 
rience accidents related to a lack of coordinated activ- 
ity between the various controllers. One particularly 
problematic feature of these new designs is a prolifera- 
tion of modes, where modes define mutually exclusive 

*The research described has been partly funded by NSF 
Grants CCR-9396181 and CCR-9520813 and NASA Grant 
NAG-1-1495. 


sets of system behavior. 

The new mode-rich systems provide flexibility and 
enhanced capabilities, but they also increase the need 
for and difficulty of maintaining mode awareness, 
which can lead to new types of mode-related prob- 
lems. This paper describes an approach to dealing 
with mode-confusion problems by analyzing the ex- 
ternal blackbox behavior of the automation for poten- 
tially error-inducing features. The results can be used 
to make tradeoff decisions during the early develop- 
ment stages of the system. 

While automation has eliminated some types of 
mode-awareness errors, it has also created the poten- 
tial for new types of errors. Sarter and Woods extend 
the classic definition of mode error and distinguish be- 
tween errors of commission (where an operator takes 
an inappropriate action) and errors of omission (where 
the operator fails to take a required action) [SW95]. 

The first automated systems tended to have only 
a small number of independent modes, and functions 
were associated with one overall mode setting. In ad- 
dition, the consequences of operator mode awareness 
problems tended to be minor, partly because feedback 
about operator errors was fast and complete enough 
that operators were able to recover before the errors 
caused serious problems (Rasmussen’s concept of error 
tolerance ) [Ras90]. 

Studies of less complex aircraft automation show 
that pilots sometimes lose track of the automation be- 
havior and experience difficulties with directing the 
automation, primarily in the context of highly dy- 
namic and/or non-normal situations [SW95]. Sarter 
and Woods conclude that in most cases, these prob- 
lems are associated with errors of commission, that 



is, with errors that require a pilot action in order for 
the problem to occur. This type of error is the classic 
mode error identified and defined by Norman — an in- 
tention is executed in a way that is appropriate for one 
mode but the device is actually in a different mode. 
Because the operator has taken an explicit action, he 
or she is likely to check that the intended effect of 
the action has actually occurred. The short feedback 
loops allow the operator to repair most errors before 
serious consequences result. This type of error is still 
the prevalent one on relatively simple devices such as 
word processors. 

In contrast, studies of more advanced automation in 
aircraft like the A-320 find that mode errors of omis- 
sion are the dominant form of error [SW95]. In this 
type of mode error, the operator fails to take an ac- 
tion that is required, perhaps because the automation 
has done something undesirable (perhaps involving a 
mode change) and the operator does not notice. In 
other words, the operator fails to detect and react 
to an undesired system behavior that he or she did 
not explicitly invoke. Because the mode or behavioral 
changes are not expected, the operator is less likely 
to pay attention to the relevant indications (such as 
mode annunciations) at the right time and detect the 
mode change or undesired behavior. 

Errors of omission are closely related to the role 
change of the operator from direct control to monitor, 
exception handler, and supervisor of the automation. 
As these roles change, the operator tasks and cognitive 
demands are not necessarily reduced, but instead tend 
to change in their basic nature. The added or changed 
cognitive demands tend to congregate at high-tempo, 
high-criticality periods [SW95]. While some types of 
errors and failures have declined, new error forms and 
paths to system breakdown have been introduced. 

Some of these new error forms are a result of mode 
proliferation without appropriate support. Providing 
support has been complicated by some unexpected 
changes in operator behavior in working with com- 
plex automation. For example, during long periods of 
flight, pilots do not have to monitor the mode annun- 
ciations continuously. Instead, they need to predict 
the occurrence of mode transitions in order to attend 
to the right indications at the right time. A-320 pilots 
have identified this new type of monitoring behavior 
in surveys conducted by Sarter and Woods. However, 
the automation and interfaces have been designed as- 
suming conventional monitoring. 

Simply calling for systems with fewer or less com- 
plex modes is unrealistic: Simplifying modes and au- 
tomation behavior often requires tradeoffs with in- 


creased precision or efficiency and with marketing de- 
mands from a diverse set of customers [SW95]. How- 
ever, systems may exhibit accidental complexity where 
the automation can be redesigned to reduce the po- 
tential for human error without sacrificing system ca- 
pabilities. Where tradeoffs with desired goals are re- 
quired to eliminate potential mode confusion errors, 
hazard analysis may be able to assist in providing the 
information necessary for appropriate decision mak- 
ing. 

To identify and evaluate potential tradeoffs, we 
need to understand why the problems occur. Acci- 
dents in high-tech systems are related to complex- 
ity and coupling [Per84, Lev95]. Perrow distinguishes 
between accidents caused by component failures and 
those, which he calls system accidents, that are caused 
by interactive complexity in the presence of tight cou- 
pling. High-technology systems are often made up 
of networks of closely related subsystems (some of 
which may involve humans). Conditions leading to 
accidents emerge in the interfaces between subsystems 
and in their interactions, and coupling causes distur- 
bances to progress from one component to another. 
Computers have exacerbated the problems by allowing 
new levels of complexity and coupling with more inte- 
grated, multi-loop control in systems containing large 
numbers of dynamically interacting components. In- 
creased complexity and coupling make it difficult for 
the designer to consider all the system hazards, or 
even the most important ones, or for the operators to 
handle all normal and abnormal situations and distur- 
bances safely. 

Some of the increased complexity has been the re- 
sult of what Sarter, Woods, and Billings have called 
technology- centered automation [SW95]. Too often, 
the designers of the automation focus on technical as- 
pects and do not devote enough attention to the cog- 
nitive and other demands on the operator. Software 
engineers building embedded controllers are rarely 
taught or understand the set of cognitive processing 
activities associated with maintaining situation and 
mode awareness and how their designs can affect these 
human activities. Instead, they tend to focus on the 
mapping from software inputs to outputs, on mathe- 
matical models of required functionality, and on the 
technical details and problems internal to the com- 
puter. Little attention has been given to evaluating 
software in terms of whether it provides transparent 
and consistent behavior that supports operators in 
their monitoring and control tasks. In fact, the pri- 
mary focus in software engineering and in artificial 
intelligence has been on producing automation that 



can function autonomously and not on supporting co- 
operation and communication between humans and 
computers. 

The result of technology-centered automation has 
been what Wiener calls “clumsy automation.” If it is 
true that mode-related problems are caused by clumsy 
or poorly designed automation, then changing the hu- 
man interface, training, or operational procedures is 
not the obvious, or at least the only solution: “Train- 
ing cannot and should not be the fix for bad design” 
[SW95]. Instead, if we can identify automation design 
characteristics that lead to mode awareness errors or 
that increase cognitive demands, then we may be able 
to redesign the automation without reducing system 
capabilities. In addition, knowing the causes of in- 
creased cognitive load will make changes in training 
or interface design more effective. The approach cho- 
sen will depend upon such factors as relative costs, 
perceived effectiveness, and required tradeoffs. 

To accomplish this goal, designers need to be able 
to identify problematic design features. Our research 
goal is to identify design constraints on the automa- 
tion based on known cognitive constraints on the hu- 
man operator and engineered or natural environmen- 
tal constraints. The first step in accomplishing this 
goal is to identify the types of errors that humans make 
in highly automated systems. Using this information, 
we can analyze the blackbox behavior specified in the 
automation requirements to predict where errors will 
occur and use this information to design the automa- 
tion and the operator procedures, tasks, and interface. 
At first, we are simply going to analyze current de- 
signs, but our long term goal is to identify software 
design criteria and techniques that will help to create 
better designs from the beginning. 

For our proposed analysis approach to work, hu- 
man errors must be non-random. After studying acci- 
dents and incidents in the new, highly automated air- 
craft, Sarter and Woods have concluded that certain 
errors are predictable [SW95]: They are the regular 
and predictable consequences of a variety of identi- 
fiable factors. Although they are “accentuated” by 
poor interface design and gaps or misconceptions in 
the user’s mental model of the system, mismatches 
between expected and actual automation behavior is 
not necessarily related to an inadequate operator men- 
tal model but can also result from inconsistent au- 
tomation behavior. Sarter and Woods identify some 
of these error forms. Degani has also identified some 
features that lead to mode confusion [Deg96], and Jaffe 
[JL89, Lev95] has identified general requirements com- 
pleteness criteria to eliminate some types of human- 


computer interaction errors. 

We want to build on the work of Sarter and Woods, 
Degani, and Jaffe to find the factors or “predictable 
error forms” that relate to automation design and de- 
vise ways to identify these factors in software require- 
ments specifications. Our approach is to model soft- 
ware blackbox behavior and provide analysis meth- 
ods and tools to search the models for predictable er- 
ror forms. In addition to providing design guidance, 
this approach might provide a way of “measuring” or 
evaluating the cognitive demands involved in work- 
ing with specific automated devices. Hansman has 
suggested that automation complexity be defined in 
terms of the predictability of the automation behav- 
ior [Hans97]. This predictability can potentially be 
evaluated using our approach. 

Analyzing designs requires an appropriate model- 
ing and specification language. This language must 
be both formally analyzable and readable without ad- 
vanced mathematical training. While automated tools 
may be necessary to analyze some aspects of large 
and complex models, we believe (and our empirical 
evidence supports the view) that the most important 
errors will be found by human experts [MLRPS97]. 
Therefore, one of our goals in the design of our model- 
ing language and tools is to provide support in human 
navigation and understanding of complex models and 
specifications. In addition, any potential design flaws 
detected by automated tools will need to be evaluated 
by humans. Thus, readability of the models is also a 
requirement for human processing of the analysis re- 
sults. Finally, the economics of system development 
are unlikely to allow for special formal models to be 
built. Instead, our analysis tools work directly on sys- 
tem and software requirements specifications. 

In the following sections, we define the concept of 
a “mode” more carefully, describe our modeling lan- 
guage, describe criteria for detecting some types of 
mode ambiguity, and demonstrate how these criteria 
might be used in analyzing the blackbox behavior of 
the automation. The language and analysis are illus- 
trated using a model of a NASA robot built to service 
tiles on the Space Shuttle. 

Definition of a Mode 

A mode defines a mutually exclusive set of system 
behaviors. One convenient way to describe behavior 
is to use state machine models. A machine or sys- 
tem can be thought of as having a set of states. The 
behavior of the system can be described by the possi- 
ble transitions from one state to another. Those state 
transitions are triggered by events, conditions, or sim- 
ply the passage of time (which can be thought of as an 



event). As an example, the following table shows the 
possible transitions between states given two system 
modes: startup mode and normal operation mode: 



a 

b 

c 

d 

e 

Startup 

c 

b 

d 

e 

a 

Normal 

c 

d 

a 

e 

a 


Table 1: A simple state machine with two modes 

The startup and normal processing modes in this 
machine determine how the machine will behave. For 
example, if the conditions occur that trigger a transi- 
tion from state c, the machine will transfer to state d 
if it is in startup mode or to state a if it is in normal 
processing mode. 

A basic tenet of linear control theory is that ev- 
ery controller contains a model of the general behav- 
ior and current state of the controlled system. This 
model may be embedded in the control logic of an 
automated controller or in the mental model of a hu- 
man controller. The model is updated and kept con- 
sistent with the actual system state through various 
forms of feedback from the system to the controller. 
When the controller’s model of the system diverges 
from the actual system state, erroneous control com- 
mands (based on the incorrect model) can lead to an 
accident [Lev95]. The situation becomes more compli- 
cated when there are multiple controllers because the 
models of the various controllers must also be kept 
consistent. A pilot, for example, must not only have 
a valid model of aircraft behavior but must also have 
a model of the automated systems’ behavior in order 
to monitor or control the automation as well as the 
aircraft. 

Mode confusion errors result from divergent con- 
troller models. See figure 1. Note that there are sev- 
eral sources of inconsistency due to improper feedback. 

In attempting to categorize factors that predict 
mode errors, it is useful to distinguish between dif- 
ferent types of modes. Degani classifies modes into 
three types [Deg96]: 

1. Interface modes specify the behavior of the inter- 
face. They are used to increase the size of the 
input or output space. 

2. Functional modes specify the behavior of the var- 
ious functions of the machine. 

3. Supervisory modes specify the level of interaction 
or supervision (manual, semi-automatic, or auto- 
matic). 


We also define three types of modes, but classify them 
differently. The modes are defined with respect to the 
control component being specified: 

1. Supervisory modes determine who or what is con- 
trolling the component at any time. Control loops 
may be organized hierarchically, with multiple 
controllers or components, each being controlled 
by the layer above and controlling the layer be- 
low (see Figure 1). In addition, each component 
may have multiple controllers (supervisors). For 
example, a flight guidance system may be issued 
direct commands by the pilot(s) or by another 
computer that is itself being supervised by the 
pilot(s). The robot motor controller (MAPS) de- 
scribed in the next section can be in either manual 
supervisory mode and controlled by a human op- 
erator, or it can accept control instructions from 
another computer called the “planner.” Mode- 
awareness errors related to confusion in coordina- 
tion between the multiple supervisors of a control 
component can be defined in terms of these su- 
pervisory modes. 

2. Component operating modes control the behav- 
ior of the control component itself. They may 
be used to control the interpretation of the in- 
terface (Degani’s interface modes) or to describe 
its required process-control behavior. For exam- 
ple, MAPS operation may be enabled or disabled 
at any time, depending on whether it is safe for 
MAPS to move the robot. 

3. Controlled- system operating modes specify sets of 
related behaviors of the controlled system and are 
used to indicate its operational state. For ex- 
ample, the MAPS model of the robot indicates 
whether it is in a moving mode (between work 
areas), in a work mode (in a work area and ser- 
vicing tiles, during which time the robot is not 
controlled by MAPS but by the planner) or is in 
an unknown mode (which means that MAPS does 
not know whether the robot is in moving mode or 
work mode). 

The Modeling Language 

Most software errors leading to accidents can be 
traced to incorrect or incomplete specifications rather 
than to incorrect implementations. While develop- 
ing hazard analysis techniques, we have been trying 
to understand how to design specification languages 
that will facilitate analysis (by both humans and au- 
tomated tools) of system and software requirements 




Figure 1: An example of a simple multiple-controller process-control system. To simplify the diagram, we have 
shown only one digital (computer) controller. In complex systems there may be several human, digital, or analog 
controllers at each level of hierarchical control and also more hierarchical levels than shown here. Note that each 
controller has several mental or logical models of the machine or process it is controlling as well as its interfaces. 
These models must be kept consistent for correct and safe monitoring and control. 














specifications. We have found that effective error de- 
tection requires specifications that are readable and re- 
viewable by human designers and application experts 
as well as analyzable by automated tools. 

Our first language, RSML (Requirements State Ma- 
chine Language), was designed while specifying the 
system requirements for TCAS II, an airborne aircraft 
collision avoidance system, for the FAA [LHHR94]. 
Using the lessons learned from this experience and 
others, we are designing a toolkit called SpecTRM 
(Specification Tools and Requirements Methodology) 
that includes a requirements specification language 
SpecTRM-RL. Underneath SpecTRM-RL there is a 
formal state machine model called RSM [JLHM91] 
upon which we have defined a set of correctness and 
completeness design criteria for safety-critical process- 
control system specifications. 

One of our goals for SpecTRM-RL is to incorporate 
features to assist in designing less error-prone human- 
computer interactions and interfaces and in detect- 
ing potential communication problems, such as mode 
confusion. Although the notation has changed from 
RSML, the system behavior is still represented using 
hierarchical and orthogonal state machines. Because 
the majority of the errors and difficulty in reviewing 
our TCAS II model stemmed from the use of internal, 
broadcast events, we have eliminated this feature. We 
have also included features to assist in finding common 
dangerous omissions and errors in process-control re- 
quirements specifications. The language design is not 
quite complete, but Figure 2 shows part of an exam- 
ple specification for a NASA robot built to service tiles 
on the Space Shuttle. The software requirements are 
taken from a master’s level project at the CMU Soft- 
ware Engineering Institute for part of a robot that 
was being designed and constructed in the Robotics 
Department [MMR92]. The system component used 
as the example in this paper is called MAPS (Mobility 
and Positioning Software). 

Because our specifications are blackbox, they must 
describe the required behavior of the component (in 
this case MAPS) in terms only of inputs, outputs, the 
relationship between these, and a model of the con- 
trolled system. The specifications do not include any 
information about the implementation or internal de- 
sign of the component, simply the input to output 
function it computes specified in terms of operating 
modes, an internal model of the controlled system, 
and an internal model of the interfaces with its super- 
visor (s) and the controlled process(es). 

As with many complex control systems, this robot 
has multiple controllers and multiple levels of control. 


MAPS is a a mid-level controller responsible for is- 
suing movement commands to the motor controller, 
which controls the mobile base of the robot (see Fig- 
ure 3). 

MAPS in turn can be controlled either by a hu- 
man operator or by an onboard computer called the 
planner. The operator controls robot movement and 
positioning using a hand-held joystick. The planner 
can also control robot movement but does so by pro- 
viding MAPS with a specification of the desired des- 
tination and route. Thus there are two supervisory 
modes: joystick and planner (see Figure 2). Either the 
human controller or the planner may assume control 
at any time, but the human controller is responsible 
for supervising the behavior of the robot at all times 
to prevent accidents, even when it is under planner 
control. Because of the distributed control structure, 
multiple possibilities for mode confusion exist. 

The supervisory interface consists of the controls 
by which a supervisor directs the control component 
(in this case MAPS) and the displays by which the 
component relays information back to the supervisor. 
(Note that displays are not limited to visual displays; 
they can also include aural and other types of com- 
munication.) The operator can control MAPS using a 
joystick with two buttons and a keyboard, as shown in 
Figure 2. MAPS provides information to the operator 
via a graphical user interface. The MAPS behavioral 
requirements use only information about the content 
of the interface, not the specific layouts or design of the 
controls and displays (which is specified elsewhere). 
The communication interface with the planner is spec- 
ified similarly. 

In addition to the supervisory interface, there is 
an interface with the controlled system (other robot 
components), which includes the inputs and outputs 
between MAPS and the various sensors and actua- 
tors. These interface models are simply the view that 
MAPS has of the interfaces — the real interface(s) may 
contain different information due to various types of 
incorrect design or failures. By separating the as- 
sumed interface and the real interface, we are able 
to model and analyze the effects of various types of 
errors and failures. 

The MAPS operational modes are: 

• enabled or DISABLED: MAPS operation is en- 
abled only if the safety circuit has signalled that 
the robot is in a safe state, the operator has de- 
pressed the deadman switch, and the robot’s ma- 
nipulator arm is stowed. 

• OFF or OPERATIONAL: MAPS may be turned off 
or it may be operational. 
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Figure 2: A partial specification of MAPS 
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Figure 3: A high-level view of MAPS 


The MAPS operational modes are relatively simple; 
a typical flight management component has a large 
number of such modes, leading to more potential for 
mode confusion. 

The controlled system (in this case the robot) is 
described (within the MAPS model) in terms of its 
operating modes and a model of its relevant states. 
The robot controlled by MAPS is either moving or 
STOPPED (performing inspection and maintenance) or 
its operating mode is unknown. MAPS only controls 
motion; the servicing of the tiles is controlled by the 
planner. The robot is also either IN- A- work- area, 
BETWEEN-WORK-AREAS, Or its location is UNKNOWN. 
The commands that MAPS issues will depend on these 
operating modes. 

The last section of the MAPS high-level specifica- 
tion is the internal model of the state of the robot. 
Note that the interface and robot models are simply 
the internal models that MAPS has of the assumed 
state of the interface controls and displays and the as- 
sumed state of the robot, not their actual, physical 
state. The internal model may not be consistent with 
the real interface and robot states due to various types 
of errors and failures. For MAPS, the physical com- 
ponents of the robot that need to be modeled in order 
to specify the control algorithm are the stabilizer legs, 
the safety circuit, the manipulator arm, and the motor 
controller. Hierarchical control is common in complex 
systems: In this case, MAPS provides commands to 
the motor controller, which itself has operating modes 
and a state model. 

The language enforces certain constraints to pre- 


vent design features that are known to lead to acci- 
dents. For example, in SpecTRM-RL, all components 
of the controlled system model (e.g., the robot model) 
must have an UNKNOWN state, which is the default for 
startup and for transitions from any type of temporary 
or partial shutdown to normal processing. The con- 
trolled system can usually continue to change state 
when the computer is shut down, and the software 
model of the process must be updated at startup or 
restart to reflect the actual process state. Many ac- 
cidents have occurred in systems where the software 
assumed the status of the process had not changed 
since the computer was last operational and issued 
commands based on this erroneous information. 

As with all state-machine models, transitions are 
governed by external events and the current state of 
the modeled system. In SpecTRM-RL, the conditions 
under which transitions are taken are specified sep- 
arately from the graphical depiction of the state ma- 
chine. We have found that the behavior of real systems 
is too complex to write on a line between two boxes. 
Instead we use a form of logic table we call and/or 
tables. Figure 4 shows an example specification of a 
transition. 

Once a blackbox model of the required system be- 
havior has been built, this model can be evaluated as 
to whether it satisfies design criteria that are known 
to minimize errors and accidents. 

Design Criteria and Analysis 

Jaffe [JLHM91] originally identified 26 complete- 
ness criteria for requirements specifications, which we 
have now extended to close to 50 criteria [Lev95]. 
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Figure 4: An example of a transition definition in SpecTRM-RL. The supervisory mode transitions from planner 
to JOYSTICK if the enabled state is entered or if the following three conditions are true: the operator selects 
joystick mode, the safety-circuit is in state SAFE, and the joystick is in the neutral position. 


These criteria include a mixture of absolute criteria 
as well as heuristics for finding flaws that frequently 
lead to accidents. Many of these are related to human- 
computer interaction such as providing appropriate 
feedback during graceful degradation and completely 
specifying preemption logic when multi-step operator 
inputs can be interrupted before they are complete. 

A few of the Jaffe criteria were derived from mathe- 
matical completeness aspects of the underlying formal 
RSM model, but most resulted from the experience 
Jaffe had in building such systems over a large num- 
ber of years. These lessons learned were used to define 
design criteria for the formal RSM model. In attempt- 
ing to extend the criteria (design constraints) to cover 
mode-confusion errors, we have taken the same ap- 
proach. 

Using the results of Sarter and Woods’ studies of 
A-320 accidents and incidents along with other reports 
of mode-related error and building on the five types 
of mode-confusion design features identified by De- 
gani, we have identified approximately fifteen design 
features of blackbox automation behavior not in our 
original Jaffe criteria that can lead to operator mode 
confusion or mode awareness errors. The work is in 
the preliminary stages, and the list will undoubtedly 
change as we investigate further. In this section, we 
illustrate the approach by describing a few items on 
the preliminary list and demonstrate their application 
to MAPS (where applicable). 

Applying our criteria to a complex control system 
will almost surely identify a large number of behav- 
iors that could lead to mode confusion. Getting rid 
of all such behaviors would most likely result in an 
overly simple control system that does not satisfy 


many of its goals. Instead, this information should 
be used to eliminate accidental complexity (i.e., the 
same functionality can be achieved but in a less error- 
inducing manner), to provide information for safety 
tradeoff analyses (perhaps by applying hazard analy- 
sis to the identified behaviors), and to design inter- 
faces, operational procedures, and operator training 
programs. For example, accidents most often occur 
during transitions between normal and non-normal 
operating modes or while operating in non-normal 
modes. Therefore, the non-normal mode transitions 
should be identified and have more stringent design 
constraints applied to them. 

The rest of the paper describes six of our categories 
of potential design flaws: interface interpretation er- 
rors, inconsistent behavior, indirect mode changes, op- 
erator authority limits, unintended side effects, and 
lack of appropriate feedback. Additional criteria can 
be found in [Lev95] and others will be described in 
future papers. 

Interface Interpretation Errors 

Interface mode errors are the classic form of mode con- 
fusion error: the computer interprets user-entered val- 
ues differently than intended or it maps multiple con- 
ditions onto the same output depending on the active 
controller operational mode and the operator inter- 
prets the interface erroneously. The latter is Degani’s 
two plant state, one display flaw. 

A common example of an input interface interpre- 
tation error occurs with many word processors where 
the user may think they are in insert mode but instead 
are in command mode and their input is interpreted 




differently than they intended. 

An example of an output interface mode problem 
was identified by Cook et.al. in a medical operat- 
ing room device with two operating modes: warmup 
and normal [CPWM91]. The device starts in warmup 
mode when turned on and changes from normal mode 
to warmup mode whenever either of two particular 
settings are adjusted by the operator. The meaning of 
alarm messages and the effect of controls are different 
in these two modes, but neither the current device op- 
erating mode nor a change in mode are indicated to the 
operator. In addition, four distinct alarm- triggering 
conditions are mapped onto two alarm messages so 
that the same message has different meanings depend- 
ing on the operating mode. In order to understand 
what internal condition triggered the message, the op- 
erator must infer which malfunction is being indicated 
by the alarm. 

A more complex example occurs in a proposed A- 
320 accident scenario where the crew directed the au- 
tomated system to fly in the track/flight path an- 
gle mode, which is a combined mode related to both 
lateral (track) and vertical (FLIGHT PATH ANGLE) 
navigation: 

When they were given radar vectors by the 
air traffic controller, they may have switched 
from the TRACK to the HDG SEL mode to 
be able to enter the heading requested by 
the controller. However, pushing the button 
to change the lateral mode also automati- 
cally changes the vertical mode from FLIGHT 
PATH ANGLE to VERTICAL SPEED — the mode 
switch button affects both lateral and verti- 
cal navigation. When the pilots subsequently 
entered “33” to select the desired flight path 
angle of 3.3 degrees, the automation inter- 
preted their input as a desired vertical speed 
of 3300 ft. This was not intended by the pi- 
lots who were not aware of the active “inter- 
face mode” and failed to detect the problem. 

As a consequence of the too steep descent, 
the airplane crashed into a mountain [SW95], 

Several design constraints can assist in reducing in- 
terface interpretation errors. The first is that any 
mode used to control interpretation of the supervi- 
sory interface should be annunciated to the operator 
(that is, it should be part of the displays interface in 
our modeling language). More generally, the current 
operating mode of the automation should be annun- 
ciated (should be in the displays interface) as well as 
being part of the operating modes. In addition, any 
change of operating mode should trigger a change in 


the current operating mode reflected in the interface 
(and thus displayed to the operator), i.e., the annunci- 
ated mode must be consistent with the internal mode. 
Consistency between displayed and current mode is, of 
course, an obvious design constraint and a violation al- 
most always signals an error in the requirements spec- 
ification. The first constraint should hold for almost 
all systems as well. 

Degani notes a third type of interface confusion er- 
ror that results from mapping a single input control ac- 
tion to multiple internal mode changes, depending on 
the order of the control actions. He calls this circular 
mode transitions . For example, pushing a button on a 
device with a small input interface (e.g., a watch with 
one or two buttons) will often cycle through the possi- 
ble modes, going to the next mode with the next but- 
ton push. A possible design constraint here is that if a 
control input is used to trigger a mode transition, then 
it must be associated with only one mode change, that 
is, the mapping from control inputs to mode changes 
is one-to-one (a mathematical function). Note that 
it is unlikely that one would want to require that the 
function be bijective, because that would eliminate the 
possibility of all indirect mode changes. For some sim- 
ple devices, even the constraint that the function be 
injective (one-to-one) may be impossible to enforce, 
and feedback about the current mode is the only pos- 
sible solution to the problem. 

Another design constraint related to these types of 
interface interpretation errors is that interpretation of 
the supervisory interface should not be conditioned on 
modes (an example is the accident related to the inter- 
pretation of “33” described earlier). This constraint 
is much stronger than the first three and may not al- 
ways be feasible or desirable to enforce. However, our 
analysis tools will highlight these transitions to the 
designer/analyst so that appropriate scrutiny can be 
applied to that part of the design. Degani’s circular 
mode transition is a subcase of this design constraint. 

In the MAPS design, while MAPS movement is be- 
ing supervised by the automated planner, the opera- 
tor is removed from process control and acts simply 
as a safety monitor. There are six conditions under 
which MAPS will stop the movement of the robot: 
(1) the robot reaches the work area, (2) MAPS is dis- 
abled, (3) MAPS enters planner mode, (4) MAPS en- 
ters joystick mode, (5) the safety circuit detects an 
unsafe condition, or (5) the deadman is released (Fig- 
ure 5. Three of these actions involves the operator 
directly — the selection of the joystick mode, selection 
of the planner mode, and the release of the deadman 
switch — and the operator will know why the robot 



was stopped. In a fourth case, the safety circuit sig- 
nals an unsafe state and an error message is generated 
and sent to the operator interface to indicate why the 
robot stopped. But the operator cannot differentiate 
between the other two reasons, and MAPS can enter 
the Disabled mode without indicating the reason to 
the operator. A straightforward solution is simply to 
provide additional status messages to the display. 

Inconsistent Behavior 

A more complex type of mode-confusion error, which 
is more often related to errors of omission than the 
interface errors mentioned above, is triggered by in- 
consistent behavior of the automation. Carroll and 
Olson define a consistent design as one where a simi- 
lar task or goal is associated with similar or identical 
actions [C088]. Consistent behavior makes it easier 
for the operator to learn how a system works, to build 
an appropriate mental model of the automation, and 
to anticipate system behavior. 

An example of inconsistency was detected in an 
A-320 simulator study involving a go-around below 
100 feet above ground level. Sarter and Woods found 
that pilots failed to anticipate and realize that the 
autothrust system did not arm when they selected 
TOGA (take off/go around) power under these condi- 
tions because it did so under all other circumstances 
where TOGA power is applied [SW95]. Another ex- 
ample of inconsistent automation behavior, which was 
implicated in an A-320 accident, involves a protection 
function that is provided in all automation configu- 
rations except the altitude acquisition mode in which 
the autopilot was operating. 

Consistency is particularly important in high- 
tempo, highly dynamic phases of flight where pilots 
may have to rely on their automatic systems to work as 
expected without constant monitoring. Even in more 
low pressure situations, consistency (or predictability) 
is important in light of the evidence from pilot surveys 
that their normal monitoring behavior may change on 
advanced flight decks [SW95]. 

Pilots on conventional aircraft use a highly trained 
instrument scanning pattern of recurrently sampling 
a given set of basic flight parameters. In contrast, 
some A-320 pilots explained that they no longer have 
a scan anymore but allocate their attention within and 
across cockpit displays on the basis of expected behav- 
ior. Their monitoring objective is to verify expected 
automation states and behaviors. If the automation 
behavior is not consistent, mode errors of omission 
may occur where the pilot fails to intervene when nec- 
essary: 


Note the fundamental difference between 
these two monitoring strategies. In the case 
of a standard pattern, the pilot’s attention 
allocation is externally guided while moni- 
toring on advanced aircraft requires mental 
effort on the part of the pilot who has to 
determine on his own where to look next un- 
der varying task circumstances. Based on his 
expectations, the pilot only monitors part of 
all available data. Parameters that are not 
expected to change may be neglected for a 
long time. A standard instrument scan, on 
the other hand, serves to ensure that all rel- 
evant parameters concerning airplane behav- 
ior will be monitored at certain time intervals 
to make sure that no unexpected and maybe 
undesirable changes occur [SW95]. 

In our previous design criteria and analysis tools, 
we include a check for nondeterminism in the software 
behavior, that is, we check to determine whether more 
than one transition can be taken out of a state under 
the same conditions [HL96]. But consistency in this 
case requires more than simple deterministic behavior 
on the part of the automation. If the operator provides 
the same inputs but different outputs (behaviors) re- 
sult for some reason other than what the operator has 
done (or even may know about), then the behavior is 
inconsistent from the operator viewpoint even though 
it is not mathematically inconsistent. More formally, 
inconsistent behavior results from two state transition 
functions of the form: 

ti : 5 x i Q x x — ► {s x OY 
t '2 : s x i Q x y — ► {s x 0} N 

where s € E is a state, i Q is an operator input, O is an 
output, and x and y can be states, reference values, 
supervisory interface values, etc. 

We have identified several different design con- 
straints related to various types of inconsistency. How- 
ever, there may be reasons why having such inconsis- 
tencies is necessary or reasonable. Again, our tools 
can point out such potential problems to the de- 
signer/analyst who must make the final decision about 
whether the automation should be changed. Because 
consistency may be most important during critical sit- 
uations or when the behavior is related to a safety 
design constraint, our hazard analysis tools may be 
able to assist with these decisions and our new intent 
specifications [Lev97] (a form of Rasmussen’s means- 
ends hierarchy adapted for software) can be used to 
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Figure 5: Example of the conditions under which robot movement is stopped 


trace such behavior back to its original system goals 
and safety constraints to identify any reasons for the 
specified inconsistent behavior. 

Indirect Mode Changes 

Indirect mode changes occur when the automation 
changes mode without an explicit instruction by the 
operator. Such transitions may be triggered on condi- 
tions in the controller (such as preprogrammed enve- 
lope protection) or sensor input about the state of the 
controlled system (such as achievement of a prepro- 
grammed target or an armed state with a preselected 
mode transition). 

Like many of the other mode-confusion problems 
noted in this paper, indirect mode transitions create 
the potential for mode errors of omission and of inad- 
vertent activation of modes by the operator. Again, 
the problems are related to changes in scanning meth- 
ods and difficulty in forming expectations of uncom- 
manded or externally triggered behavior. 

Behavioral expectations are formed based on the 
operators’ knowledge of input to the automation and 
on his or her mental model of the automation’s de- 
signed behavior. Gaps or misconceptions in the op- 
erator’s mental model may interfere with predicting 
and tracking indirect mode transitions or with under- 
standing the interactions between different modes. 

An example of an accident that has been attributed 
to an indirect mode change occurred while an A-320 


was landing in Bangalore. In this case, the pilot se- 
lection of a lower altitude while the automation was 
in the altitude acquisition mode resulted in the 
activation of the OPEN DESCENT mode. It has been 
speculated that the pilots did not notice the mode an- 
nunciation because the indirect mode change occurred 
during approach when the pilots were busy and they 
were not expecting the change [SW95]. Another exam- 
ple of such an indirect mode change in the A-320 au- 
tomation involves an automatic mode transition trig- 
gered when the airspeed exceeds a predefined limit. 
For example, if the pilot selects a very high vertical 
speed that results in the airspeed decreasing below 
a particular limit, the automation will change to the 
OPEN CLIMB mode, which allows the airplane to regain 
speed. As a final example, Palmer has described an ex- 
ample of a common indirect mode transition problem 
called a a kill-the-capture bust” that has been noted 
in hundreds of ASRS reports [Pal96]. Leveson and 
Palmer have modeled an example of this problem in 
SpecTRM-RL and shown how it could be detected and 
fixed [LP97]. 

Another example of indirect mode change can be 
found in the MAPS specification. In this scenario, 
MAPS is in joystick supervisory mode and it receives 
a message from the planner that the robot has reached 
the work area. This message will cause MAPS to tran- 
sition from ENABLED to DISABLED mode (see Figure 5) 
without any explicit instruction from the human oper- 




ator and without informing the operator of the mode 
change. If the analyst decides that this is a poten- 
tially dangerous scenario, the problem can be solved 
by augmenting the transition any — ► in-work-area 
as seen in Figure 6. 

In general, there are four ways to trigger a mode 
change: 

1. Operator explicitly selects a new mode. 

2. Operator enters data (such as a target altitude) 
or a command that leads to a mode change: 

(a) Under all conditions. 

(b) When the automation is in a particular 
state. 

(c) When the controlled system model or envi- 
ronment is in a particular state. 

3. Operator does not do anything but the transition 
is triggered by conditions in the controlled sys- 
tem. 

4. Operator selects a mode change but the automa- 
tion does something else, either because of the 
state of the automation and/or the state of the 
controlled system. 

The formal definitions are obvious and are omitted 
here. Degani also notes these types of indirect mode 
changes, but he gives them different names and clas- 
sifies them differently than we do. 

Operator errors associated with indirect mode 
changes are a phenomenon found primarily in ad- 
vanced automation. Early automation tended to in- 
volve only a small number of independent modes. 
Most functions were associated with only one over- 
all mode setting. We probably do not want to go back 
to automation that will change mode only in response 
to direct operator input, but design constraints are 
desirable that limit such indirect transitions and elim- 
inate it when possible. Our analysis methods highlight 
mode changes that are independent of direct and im- 
mediate instructions from human supervisors, and our 
tools may also be able to assist the analyst in identi- 
fying the most hazardous indirect mode changes. 

Operator Authority Limits 

Interlocks and lockouts are often used to ensure safety. 
Interlocks are commonly used to prevent hazardous 
system states by enforcing correct sequencing of events 
or actions or to isolate two events in time. A lockout 
makes it impossible or difficult to enter a hazardous 
state. 


Authority limiting is a type of lockout or interlock 
that prevents actions that could cause the system to 
enter a hazardous state. Such authority limitations 
must be carefully analyzed to make sure they do not 
prohibit maneuvers that may be needed in extreme 
situations. Recent events have involved pilots “fight- 
ing” with the automation over control of the aircraft 
after observing unexpected or undesirable aircraft or 
automation behavior. 

Various types of authority limits are used to pre- 
vent operator error or to provide protection when the 
operator cannot or does not take proper action. For 
example, automation on advanced aircraft often has 
the ability to detect and prevent or recover from pre- 
defined unsafe aircraft configurations such as a stall. 
Once a hazardous state is detected, the automation 
has the power to override or limit pilot input. 

Some accidents and incidents in highly automated 
aircraft have involved pilots not being able to over- 
come the protection limits or the pilots not being 
aware that the protection functions were in force. For 
example, the pilots during one A-320 approach dis- 
connected the autopilot while leaving the flight direc- 
tors and the autothrust system engaged. Under these 
conditions, the automation provides automatic speed 
protection by preventing the aircraft from exceeding 
upper and lower airspeed limits: 

At some point during the approach, after 
flaps 20 had been selected, the aircraft ex- 
ceeded the upper airspeed limit for that con- 
figuration by 2 kts. As a consequence, the 
automation intervened by pitching the air- 
plane up to reduce airspeed back to 195 
kts. The pilots, who were not aware that 
the automatic speed protection was active, 
observed the uncommanded automation be- 
havior. Concerned about the unexpected 
reduction in airspeed at this critical phase 
of flight, they rapidly increased thrust to 
counterbalance the automation. As a conse- 
quence of this sudden burst of power, the air- 
plane pitched up to about 50 degrees, entered 
a sharp left bank, and went into a dive. The 
pilots eventually disengaged the autothrust 
system and its associated protection function 
and regained control of the aircraft [SW95]. 

Various design criteria are related to authority lim- 
its. For example, information about any modes or 
states where the operator input is ignored or limited 
must be provided in the supervisory interface. In ad- 
dition, the analysis tools can examine the specified 
software behavior and detect exceptions to following 
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Figure 6: Modified transition to the in-work-area mode. 


operator requests. Again, the information in the in- 
tent specification is useful in determining whether such 
design features are intentional and whether they are 
related to identified hazards. 

Unintended Side Effects 

Mode ambiguity can also arise when an action in- 
tended to have one particular effect has an additional 
effect, i.e. an unintended side effect. An example oc- 
curred in the Sarter and Woods A-320 simulator study 
where it was discovered that pilots were not aware that 
entering a runway change after entering data for the 
assigned approach results in the deletion of all pre- 
viously entered altitude and speed constraints even 
though they may still apply. 

This type of design flaw differs from indirect mode 
changes in that the unintended change is not in the 
mode but in some other type of information, such as 
reference values. Degani describes this type of prob- 
lem in terms of a mode/reference value interaction, 
but more generally the same problem occurs when 
any operator entry (for example, an input value rather 
than a mode change) has unintended side effects. 

Unintended side effects can contribute to mode con- 
fusion, and often need to be evaluated by the design 
team. If a decision is made to keep the behavior, 
proper feedback constraints may be required to pre- 
vent the type of confusion that seems to result. 

Lack of Appropriate Feedback 

Many of the original Jaffe criteria or the newly defined 
criteria mentioned above are related to providing ap- 
propriate feedback (e.g., providing feedback about the 
status of interlocks and lockouts and providing grace- 
ful degradation). In general, operators need to have 
the information necessary to understand the mode 


transitions taken, i.e., the conditions that trigger tran- 
sitions. Operators need not only to track the current 
active modes and to understand their implications, 
but they also need to keep track of other automa- 
tion and system status information that may result 
in the indirect activation of modes. The difference 
between these design constraints and those requiring 
mode transition annunciations described in the section 
on interface interpretation errors is that in this case 
the automated system must not simply notify the op- 
erator that a mode change has already occurred (an- 
nunciate the present mode), but it must provide the 
information necessary for the operator to predict or 
anticipate mode changes. 

Incomplete feedback is often implicated in accident 
scenarios. For example, in the A-320 Bangalore acci- 
dent, the pilot flying (PF) had disengaged his flight 
director during the approach and was assuming that 
the pilot-not-flying (PNF) would do the same thing 
[SW95]. The result would have been a mode configu- 
ration in which airspeed is automatically controlled by 
the autothrottle (the SPEED mode), which is the rec- 
ommended procedure for the approach phase. How- 
ever, the PNF never turned off his flight director, and 
the OPEN descent mode became active when a lower 
altitude was selected. This indirect mode change (ex- 
plained above) led to the hazardous state and eventu- 
ally the accident. But a complicating factor was that 
each pilot only received an indication of the status of 
his own flight director and not all the information nec- 
essary to determine whether the desired mode would 
be engaged. The lack of feedback or knowledge of the 
complete system state contributed to the pilots not 
detecting the unsafe state in time to reverse it. 

Where automation has the ability to take au- 
tonomous actions (i.e., those not directly commanded 
by the operator), information interchange becomes 




crucial in coordinating activities and in detecting mis- 
matches between expected and actual system behav- 
ior. A behavioral description of the software, as pro- 
vided in SpecTRM-RL, is useful in determining ex- 
actly what information the operator needs to monitor 
and control the automated system. 

The problems of providing salient feedback are, of 
course, much more complicated than simply identify- 
ing the information that needs to be conveyed, but 
identification is an important step in the process. In 
our original Jaffe criteria, we identified design con- 
straints on basic feedback to the computer about the 
state of the controlled process and some types of oper- 
ator feedback requirements, but these need to be aug- 
mented with a complete set of requirements on the 
feedback to the operator or automation supervisor. 
An example constraint is that operators must have 
access to all information on critical mode transitions 
in order to predict and monitor those transitions. 

One important aspect of using feedback for error 
detection is the need for independent information. Er- 
rors can only be found through discrepancies in redun- 
dant information. One way to detect that automated 
equipment is not operating correctly is for operators to 
detect a discrepancy between the automation behav- 
ior and their mental model of how they think the au- 
tomation should work. However, operators often have 
limited understanding of complex automation behav- 
ior or are afraid to step in. 

In addition, often an error is only detectable using 
some information about the state of the environment 
or the controlled process. However, if the erroneous 
behavior is occurring because the automation is con- 
fused about the environment or system state, then it 
obviously cannot provide this information to the oper- 
ator. That is, the automation may show only consis- 
tent information because it does not know there is an 
error in its system model. Therefore, it is not surpris- 
ing that Sarter and Woods found that pilots mostly 
found errors through information given in nonauto- 
mated displays and instruments (i.e., based on obser- 
vations between desired and actual aircraft behavior, 
not on indications of the nominal status of the au- 
tomated systems). The same phenomenon is true for 
other types of systems. The problem is complicated by 
the fact that operators cannot always see what the au- 
tomation is doing and can only tell by directly observ- 
ing the reaction of the system or by getting feedback 
from some independent display. Providing indepen- 
dent feedback and providing more feedback on what 
the automation is doing can alleviate these problems. 


Conclusions and Future Work 

We have outlined an approach to reducing potential 
mode confusion errors. The software requirements are 
modeled using a hierarchical state machine language 
and then analyzed (manually or with automated as- 
sistance) to identify violations of a set of design con- 
straints associated with mode-confusion errors. The 
approach was illustrated with a model of the software 
controlling a NASA robot and a description of a few 
of our currently identified software design constraints. 

This work is still in the preliminary stages. We 
need to complete and partially validate our set of con- 
straints by examining more accidents and incidents to 
determine whether the current set would identify the 
factors involved. Once we are fairly confident about 
our list, we plan to validate the feasibility of applying 
the constraints to real specifications by building a pro- 
totype analysis tool and applying it to a model of an 
advanced aircraft FMS (probably not the A-320, from 
which many of the constraints were originally derived). 
A possible step after that would be to use incident re- 
ports from one or more of the reporting systems (e.g, 
ASRS, CHIRP, or EUCARE) for that aircraft to see 
if our predictions are accurate. 
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